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Abstract. We consider the problem of reconstructing an TV-dimensional continuous 
vector x from P constraints which are generated by its linear transformation under 
the assumption that the number of non-zero elements of x is typically limited to pN 
(0 < p < 1). Problems of this type can be solved by minimizing a cost function with 
respect to the L p -norm ||a;||p = lim e ^ + o X^tLi l a: i| p+e : subject to the constraints under 
an appropriate condition. For several p, we assess a typical case limit a c (p), which 
represents a critical relation between a = P/N and p for successfully reconstructing 
the original vector by minimization for typical situations in the limit N, P — > oo with 
keeping a finite, utilizing the replica method. For p — 1, ct c (p) is considerably smaller 
than its worst case counterpart, which has been rigorously derived by existing literature 
of information theory. 
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1. Introduction 

Compressed (or compressive) sensing is a technique for reconstructing a high 
dimensional signal from lower dimensional data, the components of which represent 
partial information about the signal, utilizing prior knowledge on the sparsity of the 
signal. The research history of this technique is rather long [H El IB]; but the horizon 
of the research field is now expanding rapidly after recent publication of a series of 
influential papers [H El El [7] . 

In a recent paper, the following issue has been considered [5]. Let us suppose a 
situation where an iV-dimensional continuous signal x e M. N is compressed to a vector 
of dimension P{< N), y G M p , utilizing a Px N signal-independent compression matrix 
F e R PxN as 

y = Fx. (1) 

We also assume that F is known and that x is sparse in the sense that the number of 
non-zero elements of x is limited to pN, where < p < 1. Then, under what conditions 
can the original signal x be correctly reconstructed from the compressed expression yl 
It is obvious that eq. flTJ in itself cannot determine a unique solution of x because 
the dimension of y, P, is smaller than that of x, N. However, the assumption on the 
sparsity of x may allow correct reconstruction. In the research on compressed sensing, 
minimization of a cost function with respect to the L p -nor nfl 

N 



\x\\ v = hm > \x., 

1=1 



\p+e 



the number of non-zero elements of x, p — 0, 

subject to the constraints of eq. ([T|) has been actively studied toward designing efficient 
reconstruction schemes exploiting such sparsity [HI El [101 E] • 

Results of [51 [71 [12] indicate that the following proposition holds. Let us suppose 
that x is an arbitrary continuous real vector the number of non-zero elements of which 
is bounded above by S. When each entry of F is an independently and identically 
distributed (i.i.d.) Gaussian random number, the probability of failure in reconstructing 
x based on the Li-norm minimization becomes arbitrarily small as iV tends to infinity 
if the inequalities 

2S, ( N\ 2S 1 fnci . 
and 

2 1/4 - 1 - M > 0, (4) 




| Eq. ([2]) does not define a norm in the mathematical sense because it violates the triangle inequality. 
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hold simultaneously. These inequalities constitute a sufficient condition for arbitrarily 
reducing the probability of failure for the Lx-based reconstruction of the arbitrary vector 
x. However, earlier studies on several other problems in information theory indicate that 
critical conditions of such worst cases [T3l[T^] are, in general, considerably different from 
those of typical cases pj)J[T6], and are not necessarily relevant in practical situations. 

This Letter is written from such a perspective. More precisely, we will herein 
assess a critical condition for successfully reconstructing x in typical cases in the limit 
N,P — > oo, but keeping a = P/N finite, utilizing methods of statistical mechanics. 
Results of numerical experiments reported in [5] indicate that a critical condition of the 
reconstruction success for typical cases is far from that of eqs. (JBJ) and (BJ. Our result 
is in excellent agreement with this indication. 

2. Problem setting 

For simplicity, we will hereinafter assume the following. Each component of the original 
signal x° G R N , x° (i = 1,2,... , N), is independently and identically generated from 
the distribution P(x) = (1 — p)5(s) +pexp(— x 2 /2) /^/2n, where p (0 < p < 1) is referred 
to as signal density and S(u) is Dirac's delta function. The compressed expression y is 
provided as y = Fx . We assume that y and F are available but x° is hidden. Each 
entry of F, F^ (p = 1, 2, . . . , P; i = 1,2,..., N), is an i.i.d. Gaussian random variable 
of mean zero and variance iV -1 . 

For generality, we formally consider a general reconstruction scheme 



utilizing a cost function with respect to the L p -norm. We will refer to eq. ([5]) as L p - 
reconstruction. In the following, we will generally examine the typical reconstruction 



compression rate a = P/N finite. 

In a recent work, utility of the L p -norm cost function in estimating x° from y + n is 
examined, where n is a zero mean Gaussian noise vector [IT] . In such problem setting, 
however, correct reconstruction of x°, which we will focus on hereinafter, is not possible 
as long as the variance per element of n is finite. 

3. Analysis 

To directly assess the typical performance of the L p -reconstruction, we have to solve 
eq. ([5]) and examine whether the solution that is obtained is identical to x° or not for 
each sample of randomly generated F and y(= Fx ). Carrying this out analytically is, 
unfortunately, difficult in practice. To avoid this difficulty, we convert the constrained 
minimization problem of eq. ([5]) to a posterior distribution of the inverse temperature 
P, thus: 




(5) 



performance for the cases of p 



0, 1 and 2 in the limit N, P 



oo, but keeping 




e -P\\ x H*6 (Fx - y) 



(6) 
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where Z(j3; y) = J dxe'^^^d (Fx — y) plays the role of a partition function. In the 
limit (3 — > oo, eq. ([6]) generally converges to a uniform distribution over the solutions of 
eq. (jSJ). Therefore, one can evaluate the performance of the L p -reconstruction scheme 
by examining the macroscopic behavior of eq. (jHJ) as /? — >• oo, for which one can utilize 
methods of statistical mechanics. 

A distinctive feature of the current problem is that eq. ([6]) depends on the 
predetermined (quenched) random variables F and x° (through y = Fx ), which 
naturally leads us to applying the replica method [IS] . Under the replica symmetric 
(RS) ansatz, this yields an expression of the typical free energy density as (3 — > oo as 

C p = - lim lim -Lpn£09;i/)] =- lim lim ^- lim -L In [Z n ((3; y)] 

Ja(Q-2m + p) QQ t XX 

= extr < h mm 

e \ 2 X 2 2 

+ (1 - p) [ Dzct> p [yfiz\ 0)+p [ Dz<P P (Vx + m 2 ^; 0) 1 , (7) 



where [• • •] represents the operation of averaging with respect to F and x°, and 
extrx{G{X)} denotes extremization of a function Q(X) with respect to X, = 
{Q, Xi m i Qi Xi m }) Dz = dz exp(—z 2 /2)/y/2rr is a Gaussian measure and 



Q) = lim <^ min ^ -x 2 - hx + \x\ p+e }}. (8) 

I a; I 2 

The term min^ {Q(X)} denotes minimization of Q{X) with respect to X. A sketch of 



the derivation is shown in Appendix A 



Three issues are noteworthy here. The first issue concerns the physical meanings of 
the variables introduced in eq. (J7J). For example, at the extremum, values of Q and m in 
eq. ((Zj) correspond to A^ _1 [(|cc| 2 )] and A^ _1 [x° ■ (x)], respectively, where (• • •) denotes 
averaging with respect to eq. (|0D as (3 — > oo and \a\ denotes the ordinary Euclidean 
norm a/X^ \ a i\ 2 f° r a vector a = (oi). This indicates that the typical value of the mean 
square error per component MSE = A^ _1 [(\x — x°\ 2 )] can be assessed as 

MSE = Q - 2m + p, (9) 

utilizing the extremum solution of eq. ([7]). When the correct signal x° dominates 
eq. (|SD as (3 — > oo, Q = m = N^ 1 [|a;°| 2 ] = p holds. Therefore, one can argue the 
typical possibility of correct reconstruction by examining whether this success solution 
dominates the extremization problem of eq. (Ej) or not. In addition, the extremized 
value of eq. ((7j), C p , itself also possesses the physical meaning of a typical value of the 
minimized L p -norm (per element) as C p = N _1 Y2iLi^ m e^+o[(\xi\ p+e )}. The second 
concerns the practical implication of eq. (181) . Comparison with analysis of the cavity 
method [19] . which is an alternative to the replica method, indicates that eq. (jSJ) stands 
for the minimization problem that a single site is effectively required to solve when the 
site is newly added to a cavity system which is defined by removing the site out of the 
original system. In that situation, the two terms Qf 2 /2 — hf constitute an effective 
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Figure 1. Profiles of x*(h;Q) for p — 0, 1 and 2. (a): Xg(h;Q) = h/Q for \h\ > ho 

and 0, otherwise, where ho — y 2Q. (b): x*(h;Q) = (h — h/\h\)/Q for \h\ > hi and 0, 
otherwise, where h\ = 1. (c): x^ih] Q) = h/(Q + 2). 



cost which arises from the constraints of eq. (pQ) by taking an average with respect 
to eq. (JH]) of the cavity system. Fig. [T] shows how the optimal solution of the right- 
hand side of eq. (jHJ) given h and Q, denoted by x*(h; Q) = —d(p p (h; Q)/dh, behaves for 
p = 0, 1 and 2. The function x*(h;Q) is of utility for constructing an approximation 
algorithm to solve eq. (jSJ) given a single instance of F and y. The final issue concerns 
the validity of the RS solution of eq. (jTJ). In applying the replica method to the 
current system, the generalized moment of the partition function [Z n (j3;y)} (n e M) 
is assessed by analytically continuing the expression of the saddle point evaluation of 

[Z n ((3;y)] = /ria=i^ ae " /3E ^ ll|a;a|lp [na=i 5 ( i71a;a -^)] forn G N to n G 1. For 
such an assessment, it is generally required to introduce an assumption about how the 
dominant saddle point behaves under permutation with respect to the replica indices 
a = l,2,...,n. For deriving eq. (I7j), we have adopted the RS ansatz, in which the 
dominant saddle point is assumed to be invariant under any permutation of the replica 
indices. However, local stability of the RS saddle point fails with respect to perturbations 
that break the replica symmetry if 



a 



X 



dxl 



+p / Dz 



dx* p (^x + m 2 z; Q) 



> 1, 



(10) 



holds, which represents the de Almeida-Thouless (AT) instability condition for the 
present problem (20]. When eq. ffTUj) holds for the extremum solution of eq. (jTJ), the RS 
treatment is not valid and one has to explore more general solutions taking the effect of 
replica symmetry breaking (RSB) into account to accurately assess the performance of 



the Lp-reconstruction. 
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4. Results 



For p — 0, 1 and 2, we numerically solved the RS extremization problem of eq. (J7J) for 
various pairs of a and p. In all cases, only a single stable solution was found. Given p 
and p, the solution found for sufficiently large a was always characterized by Q = m = p 
indicating successful reconstruction. However, as a was lowered, the success solution 
lost its local stability (against the RS disturbance) and a transition to a failure solution 
of Q 7^ m 7^ p occurred. 

For the success solution, conjugate variables Q and fh were always infinitely large 
whereas the remaining variables x an d X did not necessarily diverge. Investigating local 
stability of the success solution yielded a limit a c (p), which represented the possibility 
of .^-reconstruction in typical cases. For each of p = 0, 1 and 2, this is summarized as 
follows. 



4.1. p = 

The success solution, for which x — and x ~~ * °°> is stable if and only if a > p, which 
indicates a c (p) = p. The condition a > p is necessary to ensure that eq. ([1]) has a 
unique solution, even in the situation that all sites of non-zero elements of x are known. 
This means that the limit for p = achieves the best possible performance. However, 
due to discontinuity in the profile of Xq(Ii]Q) (Fig. CD (a)), eq. ( |T0|) always holds for 
the success solution, indicating that the current RS analysis is not valid. Therefore, 
further exploration based on various RSB ansatze is necessary for accurately assessing 
the reconstruction performance, which is, however, beyond the scope of the present 
Letter. 



4.2. p = 1 

X of the success solution is determined by 

2(1 -P) ((x + l)H(x" 1/2 )' 



X 



a 



-l/(2x)\ 

r ^ir) +p(£+1) J' 

where H(x) = J x °° Dt. Utilizing the solution of this equation, the stability condition of 
the success solution is expressed as 

a>2(l-p)H(x- 1/2 )+ P . (12) 

This indicates that the limit for p = 1 can be expressed as a c (p) = 2(1 — p)iJ(x -1 / 2 ) + p. 
a c (p) also corresponds to the criticality of eq. (fTUI) and the RS success solution is locally 
stable against perturbations that break the replica symmetry as long as eq. ( Tl2l holds. 
Therefore, our RS analysis is valid. 



4.3. p = 2 

The success solution is stable if and only if a > 1, implying a c (p) = 1. For 
a > a c (p) = 1, eq. ( fTUI) does not hold and the RS analysis is valid. Since a > 1 makes 
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Figure 2. (a): Comparison of typical reconstruction limits of the L p -reconstruction for 
p = 0, 1 and 2. Each curve denoted by represents the RS estimate of the typical 
critical compression rate a c {p) for the signal density p of the original signal for the 
Lp-reconstruction scheme. Correct reconstruction is typically possible for a > a c (p), 
but the RS estimate for p — is not physically valid due to the AT instability of 
eq. (|10[) . "L* c " (inset) represents a worst case counterpart for the Li-reconstruction 
assessed from eqs. ((3]) and (j4|) by setting P = aN and S = pN . (b): Experimental 
assessment of a c (p = 0.5) for the Li-reconstruction. Experimental data of a c (0.5, N) 
(see main text) for N — 10, 12, . . . , 30 was fitted by a quadratic function of 1/N. This 
yields a value of extrapolation a c (0.5) ~ 0.83165 for N — > oo, which is close to the 
theoretical estimate a c (0.5) = 0.83129 .... 



the constraints of eq. ([I]) sufficient to reconstruct x° perfectly, this result means that 
the L2-norm minimization is not capable of reconstructing any compressed expressions. 

Plots of the results obtained are shown in Fig. [2] (a). We also depict a curve of 
the worst case critical condition for the Li-reconstruction (inset), which is assessed 
utilizing eqs. (j3J) and (j4]) in the limit N, P — > oo, keeping a = P/N and p = S/N 
finite, for comparison. The Lx-reconstruction can be carried out in practice by interior 
point methods [21], the necessary computational cost of which grows as 0(N 3 ) in 
the present large system limit. On the other hand, performing the L -reconstruction 
is, in general, NP hard, although its potential might be superior to that of the L\- 
reconstruction. Fig. [2] (a), in conjunction with these, implies that the Lx-based scheme 
is a practically preferable method which balances computational feasibility and relatively 
high reconstruction capability. This figure also indicates that discrepancy of the values 
of critical compression rate is huge between the worst and typical case analyses. This 
implies that there may be much room for improvement of the worst case assessment 
although we must keep in mind that the criterion of reconstruction success in the 
present analysis, which permits reconstruction errors of asymptotically negligible size as 
N — > oo, is different from that of the worst case analysis, in which no errors are allowed. 

To justify our assessment, we performed extensive numerical experiments of the 
Li-reconstruction for p = 0.5, the results of which are summarized in Fig. [2] (b). In an 
experimental trial, an original signal x° was randomly generated so as to have exactly 
S = pN = N/2 non-zero elements, to which i.i.d. Gaussian random numbers of zero 
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mean and unit variance were assigned. For numerically assessing the criticality, the 
number of constraints P was lowered from P = N one-by-one until the solution of the 
Li-reconstruction, x, satisfied the condition of \\x — x ||i > 10~ 4 , and P c = P + 1 
was recorded when the condition was first satisfied. For searching for x, we used 
CVX, a package for specifying and solving convex programs [221 E3]. The trials were 
carried out 10 6 times for a fixed system size iV and the experimental critical rate 
was defined as a c (p = 0.5, N) = P c /N, where 7TT denotes the arithmetic average 
over the trials. Quadratic extrapolation from data for N = 10, 12, ... , 30 yielded an 
experimental estimate of the critical ratio a c (0.5) = linijv^oo a c (0.5, N) ~ 0.83165, 
which is in good accordance with the theoretical value a c (0.5) = 0.83129 . . . (Fig. 121(b)). 
In [5], experiments for evaluating the critical density p c for a = 0.5 were performed for 
relatively large systems of N = 512 and 1024. Judging from comparison by eye, plots of 
the results are also consistent with our theoretical estimate p c (a = 0.5) = 0.19284. . .. 
These indicate that our assessment is at least capable of explaining the experimental 
results to a high accuracy although mathematical justification of the replica method, in 
general, has not yet been established [24J. 

5. Summary and discussion 

In summary, we have assessed the typical performance of compressed sensing based 
on minimization with respect to the L p -norm for p — 0, 1 and 2, utilizing the replica 
method under the replica symmetric (RS) ansatz. Analysis of the stability condition of 
a solution which represents successful reconstruction yields a critical relation between 
the compression rate and the signal density that represents the frequency of non- 
zero elements in the original signal. We have shown that the RS solution of the 
Lo-reconstruction achieves the best possible performance, which is, unfortunately, not 
stable against perturbations that break the replica symmetry. The /^-reconstruction 
has no capability of compressed sensing. On the other hand, our RS analysis has 
clarified that the Lj-based scheme does have a considerably high reconstruction ability. 
Moreover, it has been recognized that the Li-reconstruction can be solved via linear 
programming with a feasible computational cost. These properties are advantageous 
from the viewpoint of practical utility. 

In this Letter, we have assumed that each entry of the compression matrix F is an 
i.i.d. random variable with zero mean and a fixed variance. Utilizing a technique offered 
in [25] , the analysis can be extended to cases in which F is randomly generated so as to 
be characterized as 

F T F = ODO T } (13) 

where D is a diagonal matrix, whose eigenvalue spectrum asymptotically converges to a 
fixed distribution, and O is a sample from the uniform distribution of N x N orthogonal 
matrices, independent of D. However, as long as P x P matrix FF T is typically of full 
rank, which is the case when entries of F are i.i.d. random numbers of zero mean and 
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a fixed variance, the result is identical to that obtained here (see Appendix B). One 
can also show that the values of a c (p) do not depend on details of the distribution of 
the non-zero elements of x as long as the mean and variance are finite. This implies 
that the findings of this Letter generally hold for relatively wide classes of compression 
matrices and signals. 

Performance assessment of the /^-reconstruction based on a replica symmetry 
breaking ansatz and development of mean field algorithms for approximately solving 
the reconstruction problems with a lower computational cost are currently under way. 

Note added - After submitting this Letter, the authors noticed that the typical 
criticality of the Li-reconstruction was explored in [26j [27] for compression matrices 
consisting of i.i.d. zero mean Gaussian random column vectors utilizing techniques of 
combinatorial geometry. It turns out that their weak threshold corresponds to our result 
for a c (p) with p = 1. In view of their criterion of reconstruction success, in which no 
errors are allowed, our result implies that the criticality of Li-reconstruction is "tight" 
in the sense that it does not change irrespective of whether or not we allow small errors 
which are vanishing asymptotically as N — > oo. The connection between our analysis 
and theirs further suggests a possibility of wide application of statistical-mechanics tools 
to problems in large-dimensional random combinatorial geometry. 
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Appendix A. Derivation of eq. ((7J) 



Expressions 



][[5(Fa; a 



y 



lim 



1 



a=l 



(v^FF) 

lim — — — - 

(v^FF) 



nP 



nP 



exp 



exp 



1 n 



2r 

o=l 
j P n 

fj,=l a=l 



X 



0\|2 



U, 



0\2 



(A.l) 



(A.2) 



Fx 



o 



is used and uf. 



N 



1 1 fM^i 



When are i.i.d. Gaussian random variables of 



play a key role in deriving eq. ([TJ, where y 
(/i= 1,2,. ..,P; a = 0,l,2,...,n 
mean zero and variance 1/N, which is mainly assumed in this Letter, the central limit 
theorem guarantees that can be handled as zero mean multivariate Gaussian random 
variables which are characterized by the covariances = Q a b&nv for a fixed set 

\ F denotes the operation of averaging with respect to F, 



of x°, x 1 , x 2 , 
and Q ab = Q 



ha 



X'\ 

-- N' 



where [• 

~ l x a ■ x b . 5^ is unity for p, — v and vanishes otherwise. Under the 
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RS ansatz 



Q 



ab 



Q. 

g, 

m, 
P, 



(a = b 
(a > b 
(a = 
fa = b 



1,2,. ..,n) 



1,2, 



1,2, . . . ,n,6 



n; b > a — 1, 2, . . . , n) 
0; a = 0,6= 1,2, ...,n) 



(A.3) 



0) 



this indicates that can be expressed as = ~~ Q s< ^ + ( a 



1,2, ... ,n) and 



\J p — m 2 /gs° + m/^/qtn, where and t^, are i.i.d. Gaussian random variables of 



zero mean and unit variance. Employing these expressions to eq. (1A.2I) yields 



iln 

N 



n - 2/) 



0=1 



lim — In 

N 



J Dvexp 



n(q-2m+p) ,2 
2(r+Q- g ) U 



( 27r )»/2( r + Q_ g )n/2 



-°ln 
2 



1 



n(g — 2m + p) 



(Q - g)"(27r) 



T n (Q,q,m). (A.4) 



On the other hand, the saddle point method offers an expression for the volume of the 
subshell corresponding to the RS order parameters (I A. 31) as 



1 

N 



H dx a e-w xa \\»i({x a } n a =i, Q, g, m) 



0=1 



x° 



extr 

Q,q,fh 



nQQ 



n(n 



i)gg 



— nmm + lnH n (Q, q, m; f3) 



S n (Q,q,m) 



(A.5) 



where l({x a } n a=l , a; ; Q, q, m) = []Li Wl^l 2 - AQ)^ • x a - Nm)) H a>b 5(x a ■ x b - 
Nq) and E n (Q, q, fh; /3) = lim e ^ +0 J ^Jdxexp — (Q + q)x 2 /2 + (a/^ + mx°)x 

-^1^)1,0. [• • -Lo = (1 - p) J dx°5(x°)(- ■■) + pj Dx°(- ■ •). Eqs. (El and 
(IA.5P indicate that iV _1 In [Z n {(3; y)] = extr q ^ )Tn {T n (Q, q, m) + S n (Q,q,m)} holds for 

n = 1,2, For (3 — > 00, the relevant variables scale so as to keep (3(Q — q) = Xi 

(3~ 1 {Q + q) = Q, f3~ 2 q = \ and fi~ x m = in of the order of unity. Analytically continuing 
the expression of iV -1 In [Z n (j3; y)] from n = 1,2,... to n 6 I, in conjunction with 
employment of new variables x> Qi X an d ™i yields eq. (J7j). 



Appendix B. Treatment of rotationally-invariant matrix ensembles 

For a = P/N < 1, let us suppose that compression matrix F is characterized as eq. (TIB"]) , 
where eigenvalues of diagonal matrix D asymptotically follow a fixed distribution 
r(A) = (1 - a)S(X) + ar(X) as N, P -> 00 with keeping a = P/N ~ 0(1) and O 
is sampled from the uniform distribution of N x N orthogonal matrices. This matrix 
ensemble is invariant under any rotation of coordinates. For simplicity, we assume that 
the support of r(A) is a certain finite range away from the origin A = 0, which implies 
that P row vectors of a typical sample of F are linearly independent. Under the RS 
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ansatz, employment of a formula developed in [25] to eq. (IA.10 offers an expression 

Q - q - n(Q - 2m + p) 



l[5(Fx a -y) 



a=l 



lim 

7-^+0 



G 



T 



where 



G(-x) 



extr < — 

A 



+{n-l)G 



d\r(\) ln(A + A) + 



Q-q 



not 



ln(27rr) > ,(B.l 



Aa; 



In x , 

2 2' 



(B.2) 



for x > 0. For i>1, extremization in eq. (IB. 2[) provides A ~ (1 — a)/x, which leads to 
an asymptotic form G(—x) ~ — (a/2) lax— (a/2)(l+J dAr(A) In A)— ((1— a)/2) ln(l— a). 
Utilizing this expression in assessment of eq. (IB. If) . where contributions of O(lnr) 
which arise from the G-functions are canceled with the term of — (na/2)lnr avoiding 
divergence as r — > +0, indicates that the difference between eqs. fIB.ll) and flA.4j) is only 
a constant independently of (3. This means that in the vanishing temperature limit as 
(3 — > oo free energy for the rotationally- invariant ensemble exactly accords with eq. (J7J). 
Therefore, the result is identical to that obtained in the main part of this Letter. 
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